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Observing marine mammal distribution and determining population densities is a complex problem that has been ongoing for many 
decades. Modern approaches use a varying combination of sophisticated acoustic monitoring and highly trained visual observers from 
both ships and aircraft. New electro-optical and infrared (EO/IR) camera systems with automatic detection algorithms offer the option 
to do quantitative airborne surveys safely and quickly. This paper discusses an airborne EO/IR modular system used to observe marine 
mammals, make density measurements and eventually be used to quantify animal behaviors. 


I. INTRODUCTION 

Primarily an Electro-optical and Infrared (EO/IR) system has been designed as a cost effective modular sensor for 
specific applications that can be deployed on small Tier I unmanned aerial systems (UAS’s). Over the last few years 
the series of modular sensors (EYE-5f, EYE-5g and EYE-HSswir) have been developed and tested in a number of 
applications. These modular systems will not require additional data link bandwidth nor would they add to the already 
heavy workload of the data analyst. The systems have shown that they can add value in border patrol, search and 
rescue and littoral reconnaissance operations and can make real-time detections and data products of submerged targets 
(submarines, semi-submersible, floating mines, etc.) in maritime operations. 


The system for marine mammal detection, called EYE-5f, is a coupled sensor head with Global Positioning System 
(GPS) and an image processing system. The sensor head is a turreted package containing a Long Wavelength Infrared 
(LWIR) camera for night operations, a three-channel multispectral system for animal detection and a 32X video zoom 
camera to aid in classification and identification. The processor is a Graphics Processing Unit (GPU) based system 
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capable of producing real-time processed imagery within 100 milliseconds of acquisition. Field data collections using 
this system and resulting quantitative calculations are presented. 


Visual surveys from ships are generally done using standard line transect techniques. Transect lines are established prior 
to the survey to uniformly cover a predefined study area. Marine mammals are spotted by a team of observers using 
both standard (7X) and high power (25X) binoculars. When the animal or group is seen, the location of the animal or 
group is determined by recording their relative bearing to the bow of the ship and their distance using the reticules 
in the binoculars. From aircraft, the location of the animal is determined by using a circle-back method to determine 
the animal’s location. In both ship and aircraft surveys, environmental conditions are routinely recorded to adjust for 
visibility conditions in density calculations. Quantitative density and abundance estimates are made using ‘DISTANCE 
sampling’ methodology, which accounts for the portion of the animals that were not detected near the track line and are 
not on the surface. During a recent field test in Hawaii, it was demonstrated that using EYE-Sf, animals were detected 
in excess of 40 meters in depth (clear water) in sea states of 5 and greater. This both increases the number of animals 
detected (reducing uncertainty) and reduces the sensitivity of probability of detection due to environmental conditions 
when compared to visual techniques. 


II. IMAGING AND PROCESSING SYSTEMS 

The EYE-5f modular multispectral imaging system designed by Advanced Coherent Technologies (ACT), LLC is 
shown in Fig. 1. The system dimensions are presented in Table 1. It is composed of three multispectral cameras with 
narrowband interference filters optimized to allow the system to detect marine mammals beneath the water surface, a 
Sony 32X zoom video camera and a 640x480 pixel long wave infrared (LWIR) camera. These cameras are integrated 
into a Cloud Cap Tase300 turret. The turret enables the system to scan laterally to 40° from nadir on either side of the 
aircraft as it flies. As it scans, the multispectral cameras search for whales in the water column down to depths as deep 
as 50 meters in clear water. Once detected, the system can lock on and track the animal until it surfaces, where high- 
resolution pictures from the zoom video system can be used by the observer to identify species and possibly photo 
identify individuals. The LWIR system can also be used at night to detect blowing or breaching animals off nadir, where 
the emissivity of the animal is much greater than the water. Computational power for EYE-5f system is provided by the 
Vision4ce© General-purpose Ruggedized Integrated Processor (GRIP) Delta and is also shown in Fig. 1. 


Fig. 1 — EYE-5f in display mount on the left and data acquisition/processing system Vision4ce Grip 
Delta on the right 
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Table 1 - EYE-5f sensor dimensions 


Overall System 
15 2 x 178 x 229 mm (6 x 7 x 9 inches) 
Turret Diameter: 178 mm (7.0 inches) 


Weight: Under 100 Ibs 


The GRIP Delta is a Commercial Off-The-Shelf (COTS) rugged computer system specifically designed for applications 
that can take advantage of GPU processing. The basic hardware for the system includes an Intel© Core 2 Duo Processor, 
4GB of system memory, a 256GB Solid-State Drive (SSD) for storage and an NVIDIA® GT240M graphics processor. 
Additionally, there is expansion space available, via PCle slots, that allow for additional storage, standard input/output 
(I/O), or custom I/O ports such as Camera Link, HD video, etc. The COTS design allows ACT to easily upgrade any 
and all components as they become available and ensure that there is robust support for all of the hardware featured in 
the system. 


The EYE-5f system outputs three data streams from five cameras. The data streams from the LWIR camera and the 
Sony zoom video camera are formatted for both a video tracker and a downlink. Further processing on the ground is 
possible if the downlink is available. The third is the multi-camera data stream that is transferred from the turret to 
the processor with wideband optical fibers. This multispectral data stream requires a series of processing steps. These 
steps are necessary to exploit the data so that subsurface animals are visible and automatically detected. These general 
processing steps are shown in Fig. 2. 
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Fig. 2 — Data flow diagram for general processing of multispectral data 


The addition of the NVIDIA GPU significantly increases the amount of computational power available in the system 
through the use of Compute Unified Device Architecture (CUDA)°. This architecture allows for highly parallel 
computations utilizing thousands of threads among many cores, a structure that is ideal for performing image-processing 
algorithms. These include, but are not limited to, real-time marine mammal detection, scene tracking and stabilization, 
image enhancement, or other numerically intensive vector algorithms. This provides the EYE-5f system with the ability 
to perform many of these image-processing functions onboard in real or near real time rather than in post processing. 
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Also, by placing the computations on the GPU rather than the CPU, there is typically a speedup of multiple orders of 
magnitude (algorithm dependent) as well as freeing CPU resources to deal with such tasks as image acquisition, storage, 
user input and control of any additional peripherals (such as a GPS) that might be involved. NVIDIA CUDA is also a 
COTS product and carries with it a great deal of support and documentation, making it ideal in a low cost, flexible and 
powerful image processing framework. 


HI. MARINE MAMMAL DETECTION AND DENSITIES 

ACT has been developing ways to increase the Signal-to-Noise Ratio (SNR) of whale signatures at depth with electro- 
optical systems.'” For this effort ACT has designed a detection algorithm that utilizes both the spectral and the spatial 
domains. 


Spectral Processing 

The EYE-5f in maritime application uses three multispectral cameras with narrow band interference filters, a blue band 
near 480 nm with a 20 nm bandpass, a green band near 535 nm with a 20 nm bandpass and a red band near 600 nm with a 
10 nm bandpass.’ The assumption is that the ability to see into the water column is limited by the spatially variable light 
reflected from the sea surface. This variable light is fairly wideband with blue through red light reflected from the surface 
in constant proportions. The correlation between these three bands is used in the spectral processing by subtracting a 
weighted difference between the blue band and the green band. The weighting is determined by minimizing the variance 
of the resulting image. The red band is used primarily to eliminate residual sea surface structure such as cloud shadows, 
kelp rafts, etc. 


Spatial Processing 

In the spatial domain the processing utilizes two main components: an inverse covariance matrix and a signal vector. 
The inverse covariance matrix is used to build an understanding of the spatial relationship within the dimensions of the 
signal vector. The signal vector is looking for a grouping of target pixels, depicted in Fig. 3 indicated by the 1’s in the 
circled area. Combining this information into a spatial matched filter and applying it to the spectrally processed data 
results in maximizing the SNR of the imagery. 


Fig. 3 — Graphical representation of the signal vector 
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Whale Detection Algorithm 

The whale detection algorithm can best be described by watching how an image transforms as it goes through these 
different steps. The image used for Figs. 4 through 7 was taken just west of Maui, Hawaii, in February 2009 at an 
altitude of 2000 feet. The configuration used for the multispectral cameras was a blue band (485 nm) and a green band 
(529 nm) with 23 mm lenses. There is a large difference in light levels between the center of the image and at the edges 
before the processing as shown in Fig. 4(a). This is primarily due to the lens and sun effects. It has been estimated 
that this is a quadratic trend in the image and a least-squares estimator is used to remove this effect. This technique is 
effective and practical because it extracts the data necessary from the image with no need for a priori information. An 
example of the estimated trend can be seen in Fig. 4(b). 


100 200 300 400 500 600 700 &00 900 


Fig. 4(a) — Unprocessed image Fig. 4(b) — Estimated quadratic trend 


Figure 5 shows the image after the trend has been removed. Notice how the edges are no longer dark. This sets up the 
detector with a great understanding of what to expect. The algorithm will now be able to quantify how much deviation 
from normal there is for a whale like object and classify it as a target or not. 
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Fig. 5 — Trend removed 
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The spectral process performed requires the trend to be removed from both the blue and the green band. Once this is 
done, the spatial process is applied and is shown here as a step in the overall process. Notice in Fig. 6 that a majority of 
the sea surface waves have been removed compared to Fig. 5. This reduction is expected since the spectral process is 
removing the sea surface clutter. A line profile was added and the image was annotated with the calculated SNR gain of 


3.5 dB associated with the spectral processing. 
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Fig. 6 — Spectrally processed image. The deep submerged whale is shown in the circled area. 


The last image-enhancing technique that is utilized is in the spatial domain. Figure 7 was created using a 7x7 size 
mask convolved over the image in Fig. 6. This mask is a good all-purpose filter for whales at depth and is proving very 
effective. The size of the mask can vary. It depends on a target size. Notice the whale just to the right of the main pod in 
Fig. 7, which was virtually impossible to see in Fig. 6 before the spatial processing was applied. 
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Fig. 7 — Spatial and spectral processing applied 


Estimating Marine Mammal Population Density 

All data collected used the EYE-5f system and were aimed at collecting data necessary to insure that the automatic 
detection algorithms were working properly and in real time. An overall goal was to assess the use of the EO/IR system 
to make marine mammal density measurements. In order to meet this goal, ACT developed a plan for making aerial 
surveys with the EYES system. The aerial survey process includes the following steps: 

1. Conduct a strip transect survey. The strips should be spaced to insure that an animal detected in one pass cannot 
swim to the next before the aircraft gets there. This insures no animal is counted twice. 

2. Fly tracks that are not influenced by the presence of visual detections. The purpose of this is that it is assumed 
each observation is independent and random. This allows us to get the density by simply dividing the number 
of animals detected (adjusted to compensate for the dive profile and the water clarity) by the area surveyed to 
get the density. If visual observations were taken into account, the cross track probability of detection for the 
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visual observer would need to be accounted for. This would increase the complexity of the density calculation. 

3. Estimate the dive profile of the type of whales (in this case Humpbacks) to get the probability of seeing an 
animal above the depth to which the sensor can detect. While this may not be available for many species, it 
is essential information for any type of density survey and must be estimated. This reinforces the valuable 
information collected by tagging. 

4. Quantify EYE-5f’s depth capability as determined by water clarity, surface waves, sun glint, etc. 

5. Calculate the area covered during the survey. 

6. Count the number of whales detected. 


The density (D) is then given by the equation: 


N 
D = —, 1 
AP (1) 
where N is the number of whales counted, A is the area covered by the camera images and P is the probability that if a 
whale is present in the field of view it is detected. 


In March 2010, the first density survey was undertaken over the southwestern waters of Maui, Hawaii. This area is 
breeding and calving ground for Humpback whales in the winter months. During this time, the density of the animals 
is fairly high and is perfect for testing the utility of using EY E-5f to determine animal density using the electro-optical 
system. A less than two-hour collect was used to measure whale densities from an altitude between 1500 and 2000 feet. 
Altitude was limited by cloud cover and not by system capability. For large whales, the system can easily be operated 
in excess of 5000 feet in clear skies. Figure 8 shows the tracks flown in the survey. While these actual tracks are only 
generally similar to the planned tracks to be flown in the strip survey, they are still not influenced by the presence of 
animals and thus detections are still random and the survey is valid. A sea state of 6 was recorded with wind gusts greater 
than 40 knots at times. The high winds are the primary reason that the actual tracks flown do not resemble a regular, and 
planned, grid. Under these high wind conditions, visual airborne surveys would be impossible. 
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Fig. 8 — Google image of day one afternoon flight tracks for two-hour whale survey 


To build a distance sampling survey, the dive profile of a Humpback whale needs to be quantified. The information 
used can be seen in Table 1.* Table 2 is derived from Baird’s measurements of dive profiles for a limited number of 
Humpbacks in the same area as our survey. It shows the average cumulative percentage of time the animals spend above 
a specific depth. This cumulative depth distribution would technically be valid only for behaviors, sex and sizes of the 
whales represented in the Baird’s measurements. The next step is to quantify EYE-5f’s ability to detect at depth. It has 
been verified that EYE-5f can detect in excess of 1.5 diffuse attenuation lengths in Hawaiian waters (Jerlov water type 
1b under the conditions of these data collects), or approximately forty meters.” Comparing this to the dive profile of a 
Humpback whale, there is an 82% probability that EYE-5f will detect a Humpback whale in this region if one is in the 
sensor’s field of regard. 
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Table 2 - Baird’s Dive Profile of Humpback Whale in the Maui Breeding and Calving Areas 


Depth (meters) Mean % time in Depth Category 
39.55 39.55 


An example of the enhancement for detection that the ACT algorithm can provide is shown in Figs. 9 and 10. This image 
was taken with the EYE-5f system in sea state 6 and contains two Humpback whales. The whales are difficult to see in 
the preprocessed image in Fig. 9, but after the band difference minimizing-variance algorithm, shown in Fig. 10, they 
are clearly visible and ready for spatial processing. 
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Fig. 9 — Multispectral pre-processed data with two Humpbacks 
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Fig. 10 — Multispectral post-processed image data clearly showing the two Humpbacks 


The next step in the survey process is to determine the number of whales detected and the survey area covered. The 
processed survey data revealed 20 whales detected in 13 “pods”, or groups, and covered a survey area of approximately 
30 square miles. There was an average of 1.54 whales per pod. The equation for Humpback whale density was presented 
in Eq. 1. The estimated Humpback whale density for the area covered is 0.32 Humpback whales per square kilometer. 


If it is assumed that pods of whales do not interact with each other, then a Poisson distribution in pods (rather than 
individual whales) can be used to estimate uncertainty. The error in number of whales is thus the square root of the 
number of pods times the number of whales per pod, or about six whales. The density is thus 0.32 whales (+/- 0.089) 
whales per square kilometer. In the roughly 400 square miles between Maui and Lanai and in Maui’s Ma’alaea Bay, 
there would thus be about 333 Humpback whales (+/- 92 whales) during the winter breeding / calving months assuming 
a uniform spatial distribution over this area. Either increasing the swath or increasing the amount of time taken to 
conduct the survey can minimize the error of estimation. Both could easily be done. It should be noted that even with the 
narrow field of view of the EYE-5f multispectral system, a survey covering 30 square miles takes less than two hours. 


Over the past several years, the Pacific Whale Foundation (PWF) has been organizing volunteers to count the number 
of Humpback whales visibly in the same area of the EYE-5f airborne surveys. The sighting methodology is consistent 
from year to year so the counts can reflect relative population changes over time. The sighting average over the last few 
years is about 1300, with 1208 sightings in 2010. The 2010 counts were done just three days prior to EYE-5f surveys; 
however, their counts are conducted from survey points all around Maui and not just from Lahaina thru Ma’alaea Bay 
area as surveyed by EYE-5f. Although the sighting methodology is consistent year to year, the animals are potentially 
counted multiple times by multiple people and the data is thus not a direct density measurement. 


The aerial survey demonstrated that this type of automated system could be very useful in quantitatively measuring 


marine mammal densities quickly when the species densities are reasonably high. For an accurate survey, it can be 
assumed that at least 20 detections are needed. More detections decrease the numerical uncertainty, and less detections 
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increase it. During the survey, the system did not scan, so the field of regard was just 10° either side of nadir. If scanning 
is employed, the field of regard can be increased by a factor of four to plus and minus 40 degrees from nadir and still 
get the same detection depth performance. In addition, the altitude can be doubled to 4000 feet easily and still have 
the resolution needed for detecting animals as small as dolphins. Those two factors coupled with an increase in aircraft 
speed gives an order of magnitude increase in area search rate; meaning that in the same two hour survey, 200 whales 
would have been sighted, or 20 whales would have been sighted if the density was 10 times smaller. Ultimately the 
time required to conduct a survey depends on the density of the animal population and the acceptable error. The EYE-S5f 
system is designed to make the best use of this time, so that it can operate in relatively low densities and still provide 
accurate findings. 


IV. MARINE MAMMAL BEHAVIOR 

Overall uncertainties are again dominated by the uncertainty of the marine mammal distribution model. In this case, 
proper dive profiles and possibility of seeing the whales at depth are again very important to correctly normalize the 
counts. The designed detection algorithm has enhanced the image to the point where targets below the sea surface show 
up that were not previously seen. The following examples highlight the fact that the EYE-5f multispectral imaging 
system can see beneath the surface. This capability has many implications, including the potential to improve the 
accuracy of distance sampling surveys. 


Depth of Whales 
The algorithm utilizes a depth equation that uses the contrast difference between a whale at the surface and a whale at 
depth. The equation is: 


z = In[C(z)/C(0)|/(2k), (2) 


where z is the depth of the target measured in meters, C(z) is the contrast of the target at depth z, C(0) is the contrast of 
the target at the surface and k is the downwelling irradiance diffuse attenuation coefficient with units of 1/meters. k can 
be estimated using previous water clarity measurements or eventually directly from the data itself based on the color of 
the water. This equation will provide the data necessary to determine our system’s capability to determine the depth of 
whales detected. Figures 11 through 14 are presented examples of whale depth calculations. Note that it is not necessary 
that the C(0) whale be in every image, as long as the image in which the zero depth whale contrast was measured was 
taken under the same conditions. 


Fig. 11 — Processed image of nine whales detection at depth (z) on the right with the corresponding RGB image on the left 
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Fig. 12 — Processed image of six whales detection at depth (z) on the right with the corresponding RGB image on the left 


Fig. 13 — Processed image of three whales detection at depth (z) on the right with the corresponding RGB image on the left 


Fig. 14 — Processed image of two whales detection on the right with the corresponding RGB image on the left. The processed 
image indicates a whale at depth just to the right of the obvious whale image. 
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Size Distribution and Distance Between Whales 

In order to determine a whale size distribution, images of 12 whales were taken (10 adults and 2 calves). Sizes were 
measured in meters using the ruler tool in Google earth. The average size of an adult whale is 12.9 meters, and a calf 
(baby whale) is 7.7 meters. 


Figure 15 presents an example of Google image showing one of the processed images with a pod (2 whales) of 
Humpbacks. The process uses the Euler angles of the gimbal and the aircraft along with the fixed field of view of the 
imager to geolocate the four corners of the image on the sea surface. Once geolocated, apparent sizes and distances can 
be readily measured with less than 1 meter accuracy. 
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Fig. 15 — Google image showing one of the processed images with a pod of Humpbacks. The image is clamped to the sea 
floor on Google Earth to measure the size of the whales. 


Figure 16 presents an example of image showing 3 Humpback whales and example of calculation whale depth (z), 
whale size (s) and distance between the whales (d). The images from marine mammal survey were positioned in the 
Google Earth map and clamped to the sea floor in order to measure size of whales and distance between whales. The 
whale depth, size and distance between whales are measured in meters. 
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11FEBO9 ML23p 122741 202741Z IF: 280.43 VA: 0 AA: 2000 FT 
N20 50 15.71 W156 43 34.97 NA: 102.49 CA: 90.00 SR: .33 NM 


Fig. 16 — Example of data showing whale depth (z), whale size (s) and distance between whales (d) 


All images collected during marine mammal densities survey come along with metadata, which was used to annotate 
the imagery. Definitions of the annotations are shown in Fig. 17. The annotation can be used to place the image in space 
and time as well as provide the geo location of the camera and define the collection angles; this allows further analysis 
of the image. In the marine mammal monitoring case, further analysis has included marine mammal size, depth, spatial 
separation and density. 
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Fig. 17 — Definitions of the annotation (meta data) for the image presented in Figure 16 


V. CONCLUSIONS 

A reasonably low-cost, multispectral, multichannel, turreted optical system has been presented that is suitable for use 
on small UAS platforms as well as manned fixed and rotary wing aircraft. The system has been shown to be capable of 
making quantitative observations of Humpback whales both on and near the surface of the ocean in a known breeding 
ground. When the marine mammal densities are sufficient (greater than about | animal per hundred square miles), this 
type of system can provide a quick and efficient way to remotely measure the animal density for the surveyed region. 
The system can also be used to monitor the effects of nearby ship traffic and airborne observers and, if timing is good, 
to record how the animals directly react to other nearby anthropogenic sounds. The key to monitoring behaviors is the 
ability to observe the animal over a period of time. To do this, the aircraft could easily orbit a high enough altitude to 
presumably be unnoticed by the animals and still be constantly observing and quantifying depths and relative positions. 


While this paper presents a system used only for observing the behaviors and densities of underwater animals, there are 


clearly other application for this type of system. EO/IR multispectral systems have been, and are being, used in a variety 
of maritime applications including search and rescue, mine detection and antisubmarine warfare. 
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